feat(skills): detect skill-lock drift and flag a stale hub catalog - #2929

🟡 Important

1. Pre-digest lock entries are reported as content mismatch (src/gaia/skills/cli.py:525)

LockEntry.content_digest defaults to "", so every entry written before this PR yields a DRIFT_UNRECORDED from _content_drift (drift.py:985). notable in _handle_list filters out only DRIFT_UNTRACKED, so UNRECORDED (and MISSING) fall into a message that claims the files "no longer match" — false for both kinds. Same path makes gaia skill lock exit 4 for every upgrading user with a hub install.

test_a_lock_entry_without_a_digest_reports_as_unverifiable covers check_drift for this case, but nothing covers what gaia skill list prints for it.

Cheapest correct fix is to key the wording off the drift kinds present:

    if notable:
        blocking = sum(1 for d in notable if d.fatal)
        unverifiable = [d for d in notable if d.kind == DRIFT_UNRECORDED]
        names = ", ".join(sorted({d.skill for d in notable}))
        if len(unverifiable) == len(notable):
            print(
                f"\n⚠ {len(notable)} skill(s) predate content digests, so "
                f"{drift.lock_file.name} cannot verify them ({names}). "
                "Record them once with: gaia skill lock --relock",
                file=sys.stderr,
            )
        else:
            print(
                f"\n⚠ {len(notable)} skill(s) differ from "
                f"{drift.lock_file.name} ({names}"
                f"{f'; {blocking} will refuse to load' if blocking else ''}). "
                "Details: gaia skill lock --check",
                file=sys.stderr,
            )

(needs DRIFT_UNRECORDED added to the local import on cli.py:477.) Worth a gaia skill list test that seeds a digest-less entry and asserts the wording.

2. Lock read failures now propagate through list and load (src/gaia/skills/cli.py:484, src/gaia/skills/manager.py:280)

SkillLock.load raises SkillValidationError on unreadable JSON or a schema-version mismatch (lock.py:159-179). _handle_list calls check_drift unconditionally before rendering anything, and SkillManager.load calls assert_no_fatal_drift for every ROOT_USER skill. Net effect: a truncated lock file (partial write, full disk) or a downgrade after a future schema bump means gaia skill list prints nothing at all, and a locally-authored skill with no lock entry whatsoever fails to load.

Not arguing for a silent fallback — the error should still surface. The question is whether it should replace the listing rather than accompany it. Reporting it the way manager.discovery_errors are reported keeps the failure loud and the command useful. Your call; flagging it because the blast radius is wider than the marketplace feature that owns the file.

🟢 Minor

3. Relock claims success when it re-recorded nothing (src/gaia/skills/cli.py:1046)

When every drifted entry is refused, changed is False but refused is non-empty, so the output leads with ✅ Re-recorded … and only then prints ❌ Refused to re-record ….

    if result.changed:
        print(f"✅ Re-recorded {result.lock_file}")

4. Staleness banner is ungrammatical (src/gaia/skills/cli.py:894)

age_text is a phrase ("93 days ago"), so this renders as "This catalog is 93 days ago — over 7 days old." The same broken sentence is copied into docs/reference/cli.mdx:63.

            f"⚠ This catalog was last refreshed {found.age_text} — over {days} "
            "days old. The hub "

5. relock hashes each directory twice (src/gaia/skills/drift.py:1286)

_inspect computes content_digest(directory) to detect drift, then line 1301 computes it again to store it. Fine for small skills, wasteful for one with a large resources/ folder.

6. Agent.load_skill's Raises: list is now incomplete (src/gaia/agents/base/agent.py:1182)

It enumerates SkillNotFoundError / SkillValidationError / SkillPermissionError; SkillDriftError can now propagate through it too.

Strengths

• --relock refuses to launder an attestation. The obvious implementation would rewrite the digest on request; refusing for community/verified and offering the two honest exits (reinstall, or remove + re-import at experimental) is the part that makes the feature mean anything. test_relock_refuses_to_re_record_an_attested_skill_that_drifted asserts the skill still refuses to load afterwards — the check that catches a regression here.
• Fixtures run the real publish → install path. Lock entries in test_skills_drift.py come from Marketplace.publish / .install against the stand-in hub, so if install_skill stopped recording the digest the tests fail. A hand-built LockEntry fixture would have kept passing.
• forget_skill is wired into all three replace paths (import, create --force, migrate --force), which is what stops a local replacement from reading as tampered-with hub content — the failure mode this feature would otherwise have introduced.
• Reusing audit.findings.content_digest gives one content identity across audit, publish, and install, and inherits its __pycache__/.pyc exclusions — so importing a skill's tools.py can't trip a false fatal drift.
• The docs land in all three places the claim lives (CLI reference, skills spec, format plan), including the honest note that skill-lock.json has no integrity protection of its own.

1. Pre-digest entries reported as mismatch — fixed. _handle_list splits notable into DRIFT_UNRECORDED and everything else, one message each; _handle_lock prints a predate content digests header when every non-UNTRACKED drift is UNRECORDED. I excluded UNTRACKED from that test rather than requiring the whole report to be unrecorded — one leftover create/import directory would otherwise have flipped the header straight back to "difference(s)".

gaia skill lock still exits 4 on this state, deliberately: it is an explicit verification verb, and "could not verify" is not "verified clean". gaia skill list exits 0. Both are now stated in docs/reference/cli.mdx so nobody gates CI on it blind.

2. Lock read failures — half taken. _handle_list catches SkillValidationError from check_drift into a lock_error reported alongside the table (and as a lock_error key in --json, with drift: null); exit stays 4. SkillManager.load is unchanged, and test_a_damaged_lock_still_refuses_to_load_a_user_root_skill pins that as a decision rather than an oversight.

3–4. Relock success line, staleness grammar — fixed as suggested, plus the doc copy.

5. Double hash in relock — declined. Threading the digest out of _inspect makes the common check_drift path carry a value it discards to save I/O in a rare, manually invoked command, and introduces a staleness window between the hash that verifies and the hash that gets stored. If skill directories ever grow large enough for this to matter the fix belongs in content_digest, not its callers.

6. Agent.load_skill Raises: — fixed, and its SkillValidationError entry was stale too: since the drift check landed it also fires when skill-lock.json is unreadable, which is a different cause with a different fix.

The laundering hole. _content_drift returns DRIFT_UNRECORDED with fatal=False for an entry with no digest — correct for the load path, since bricking every attested skill on upgrade would be worse. But relock refused only on d.fatal, so it recorded a fresh digest for a community entry under its existing installed_tier and signature.signed. Verified end-to-end against the real publish → install path before fixing: updated: ['web-research'], refused: [], check_drift().fatal == (), loads clean at community.

Refusal now keys off _would_launder(entry, drift) — fatal, or unrecorded on an attested entry — and _content_drift's remediation branches on _attested so it names gaia skill install <name> --force instead of a relock that will refuse. test_relock_refuses_to_stamp_a_digest_onto_a_pre_digest_attested_entry is the regression guard; test_relock_still_records_a_pre_digest_entry_that_nothing_attested keeps the refusal scoped to attestation.

Docs. cli.mdx (upgrade path, exit codes, corrupt-lock blast radius, the widened --relock refusal), spec/agent-skills.mdx, and plans/skill-format.mdx were updated together — the relock refusal and the corrupt-lock behavior are each described in more than one of them.

Validation. black / isort / flake8 / pylint clean; 1383 skills + hub tests pass. Nine new tests, seven of which fail on the pre-fix code. No LLM-affecting path is touched, so no eval run.

src/gaia/skills/cli.py — _handle_lock() (around line 1026):

# current — no error handling for corrupt lock
report = check_drift(root)

# fix — mirror _handle_list's pattern
lock_error = ""
try:
    report = check_drift(root)
except SkillValidationError as exc:
    lock_error = str(exc)
    if getattr(args, "as_json", False):
        print(json.dumps({"error": lock_error}, indent=2))
    else:
        print(f"\n❌ {lock_error}", file=sys.stderr)
    return EXIT_INVALID

Same guard needed for the --relock path: wrap relock(root) in the same try/except before passing to _render_relock.

Test to add alongside the existing test_cli_skill_list_reports_a_damaged_lock_without_hiding_the_skills:

def test_cli_skill_lock_reports_a_damaged_lock_cleanly(
    marketplace, attested, monkeypatch, capsys
):
    monkeypatch.setattr(skills_cli, "_manager", lambda: marketplace.manager)
    (marketplace.skills_root / "skill-lock.json").write_text("{trunca", "utf-8")

    assert skills_cli.handle(_lock_args()) == skills_cli.EXIT_INVALID
    assert "not readable JSON" in capsys.readouterr().err

Why there is no traceback. handle() (src/gaia/skills/cli.py:461-471) wraps the handler dispatch in except SkillNotFoundError / except SkillValidationError / except SkillError. SkillLock.load raises SkillValidationError on both unreadable JSON and a schema mismatch, and every skills exception subclasses SkillError, so check_drift() and relock() raising out of _handle_lock lands in that arm. EXIT_INVALID is 4, which matches what docs/reference/cli.mdx promises — I checked the constant rather than trusting the number.

Real-CLI evidence (isolated GAIA_CONFIG_DIR, one experimental skill, lock overwritten per row):

lock --check   exit=4 traceback=no stdout_bytes=0 err=is not readable JSON
lock --relock  exit=4 traceback=no stdout_bytes=0 err=is not readable JSON
lock --json    exit=4 traceback=no stdout_bytes=0 err=is not readable JSON
lock --check   exit=4 traceback=no stdout_bytes=0 err=is schema v99
lock --relock  exit=4 traceback=no stdout_bytes=0 err=is schema v99
lock --json    exit=4 traceback=no stdout_bytes=0 err=is schema v99

The tests. test_cli_skill_lock_reports_a_damaged_lock_cleanly is parametrized over {truncated, future-schema} × {--check, --relock}; test_cli_skill_lock_json_does_not_emit_a_half_report_for_a_damaged_lock asserts empty stdout. Mutation-checked: removing handle()'s try/except entirely fails all five. Removing only the SkillValidationError arm does not fail them, because the SkillError arm below is identical — the property pinned is "exit 4 with an actionable message", not which clause delivers it.

tests/unit/test_skills_drift.py only; no source change. _lock_args() moved to setdefault so as_json=True can be passed without colliding with the default.

Validation. black / isort / flake8 clean on the changed file; the five pure-Python util/lint.py gates (security-suppressions, imports, agents, dependabot, doc-versions) pass. util/lint.py --all itself cannot run in this environment — it shells out through uvx, which cannot reach PyPI here — so the formatter and linter were run from the local install instead. 1218 skills + hub-catalog tests pass; the three test_real_cli_* failures are a pre-existing Windows-local env issue (the subprocess helper scrubs USERPROFILE) and reproduce on an unmodified checkout.

Not fixed, flagging it. relock() writes through SkillLock.save(), whose mkdir/write_text are unguarded — a read-only or full disk would escape as a genuine traceback. That is a real gap, it is not the one reported, and it is shared with the install path, so it belongs in its own change rather than bolted onto this branch.

src/gaia/skills/cli.py — _handle_lock (new in this push):

def _handle_lock(args: argparse.Namespace) -> int:
    from gaia.skills.drift import ..., check_drift, relock

    root = _manager().user_root

    if getattr(args, "relock", False):
        return _render_relock(relock(root), root, as_json=args.as_json)  # ← raises

    report = check_drift(root)   # ← raises

Both check_drift and relock call SkillLock.load(root) → raises SkillValidationError on a corrupt or future-schema file. _handle_list wraps the same check_drift call in try/except SkillValidationError as exc: lock_error = str(exc) — _handle_lock should do the same.

    try:
        if getattr(args, "relock", False):
            return _render_relock(relock(root), root, as_json=args.as_json)
        report = check_drift(root)
    except SkillValidationError as exc:
        if args.as_json:
            print(json.dumps({"lock_error": str(exc)}, indent=2))
        else:
            print(f"❌ {exc}", file=sys.stderr)
        return EXIT_INVALID

SkillValidationError is already imported at the top of the file.

handle() (src/gaia/skills/cli.py:468-478) wraps every handler, lock included:

try:
    return handler(args)
except SkillNotFoundError as exc:
    sys.stderr.write(f"❌ {exc}\n"); return EXIT_NOT_FOUND
except SkillValidationError as exc:
    sys.stderr.write(f"❌ {exc}\n"); return EXIT_INVALID
except SkillError as exc:
    sys.stderr.write(f"❌ {exc}\n"); return EXIT_INVALID

_handle_lock is registered in that same handlers dict (line 452), so both check_drift(root) and relock(root) land in the SkillValidationError arm.

Reproduction, via GAIA_CONFIG_DIR so it does not touch a real ~/.gaia:

export GAIA_CONFIG_DIR=$(mktemp -d)/g && mkdir -p "$GAIA_CONFIG_DIR/skills"
printf '{trunca' > "$GAIA_CONFIG_DIR/skills/skill-lock.json"
gaia skill lock;           echo "EXIT=$?"   # EXIT=4
gaia skill lock --relock;  echo "EXIT=$?"   # EXIT=4
gaia skill lock --json;    echo "EXIT=$?"   # EXIT=4, stdout empty

Repeated with {"schema_version": 99, "skills": {}} → is schema v99; this GAIA build reads v1. Upgrade GAIA, or delete the lock and reinstall your skills., exit 4.

Coverage: tests/unit/test_skills_drift.py::test_cli_skill_lock_reports_a_damaged_lock_cleanly (4 params: {check, relock} × {truncated, future-schema}) and ::test_cli_skill_lock_json_does_not_emit_a_half_report_for_a_damaged_lock. Full file: 41 passed.

• The Email Triage Eval (synthetic corpus, report mode) failure dated from Sep 8; that lane has been rewritten twice on main since (1dc04651f, ed90d9ee3), so the old result is not comparable.
• The branch's own four commits are untouched — this is a merge commit only, no rebase, no force-push.
• tests/unit/test_skills_drift.py, test_skills_cli.py, test_hub_catalog.py on the merged tree: 117 passed, 3 failed. The three are @pytest.mark.slow test_real_cli_* cases that fail on Windows only — _real_cli builds a scrubbed env with HOME but no USERPROFILE, so Python 3.13's Path.home() raises. Same helper is on main, so it is pre-existing and Linux CI is unaffected.
• rag_quality + context_retention + tool_selection vs Gemma baselines will stay red — that gate is down repo-wide while the judge is spend-capped (ci(eval): every eval gate dies in preflight — the judge org is out of monthly spend #4080), not a signal about this branch.